United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
I nilid Stall-, l'atint and Trademark Office 

Address: COMMISSIONER FOR PATENTS 



APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO. 



2()()(M)042C'on 



26652 7590 02/04/2009 

AT&T CORP. 
ROOM 2A207 
ONE AT&T WAY 
BEDMINSTER, NJ 07921 



EXAMINER 
HAJNIK, DANIEL F 



ART UNIT PAPER NUMBER 



MAIL DATE | DELIVERY MODE 

02/04/2009 PAPER 



Please find below and/or attached an Office communication concerning this application or proceeding. 

The time period for reply, if any, is set in the attached communication. 



PTOL-90A (Rev. 04/07) 



l/ffflrC? nVrliUli Otfff Iff ids y 


Application No. 

10/662,550 


Applicant(s) 

COSATTO ET AL. 


Examiner 
DANIEL F. HAJNIK 


Art Unit 

2628 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address — 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )|EI Responsive to communication(s) filed on 1 1/26/2008 . 
2a )□ This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) |EI Claim(s) 22-25,27-32,34 and 35 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) |EI Claim(s) 22-25,27-32,34 and 35 is/are rejected. 

7) Q Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) Q The specification is objected to by the Examiner. 

10) ^ The drawing(s) filed on 15 September 2003 is/are: a)^ accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

20 Certified copies of the priority documents have been received in Application No. . 

3.Q Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attach ment(s) 

1) D Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Paper No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5 ) □ Notice of Informal Patent Application (PTO-152) 

Paper No(s)/Mail Date . 6) □ Other: . 



PTOLSe^Re^T-OsT 108 



Office Action Summary 



Part of Paper No./Mail Date 20090123 



Application/Control Number: 10/662,550 Page 2 

Art Unit: 2628 

DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.1 14. Applicant's submission filed on 1 1/26/2008 has been entered. 

Response to Amendment 

1 . The affidavit filed on 1 1/26/2008 under 37 CFR 1 . 1 3 1 has been considered but is 
ineffective to overcome the Cox reference. Please see the response to arguments section below 
for more details. 

Claim Rejections - 35 USC §101 

1. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

Claims 22-25, 27-32, 34, and 35 are rejected under 35 U.S.C. 101 because the claimed invention 
is directed to non-statutory subject matter. 

Claim(s) 22-25 and 27-29 are rejected under 35 U.S.C. 101 as not falling within one of the four 
statutory categories of invention. While the claims recite a series of steps or acts to be 
performed, a statutory "process" under 35 U.S.C. 101 must (1) be tied to another statutory 
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category (such as a particular apparatus), or (2) transform underlying subject matter (such as an 
article or material) to a different state or thing. The instant claims neither transform underlying 
subject matter nor positively tie to another statutory category that accomplishes the claimed 
method steps, and therefore do not qualify as a statutory process. 

As per independent claim 22, the method, or process, of the claim describes steps to 
accomplish the claimed method, however each of the claimed steps appears to be an operation 
that may be performed manually without the use of a machine. In addition, the claim recites an 
audio/video unit in connection with the selection process, however, it is not required in the 
performing of the selection step and therefore is neither explicitly structural tied nor inherently 
involved in the step. The claim language does not require or positively recite any use of a piece 
of hardware (as one example). For example, the method synthesizes animation. However, there 
is no mention of, for example, a processor or memory to accomplish any of the claimed method 
steps. Since positive recitation of such a device or apparatus is absent in the claimed steps, the 
claim is non-statutory. 

As per independent claim 23, the method, or process, of this claim also describes steps to 
accomplish the claimed method, however each of the claimed steps appears to be an operation 
that may be performed manually without the use of a machine for the same reasons as claim 22. 

As per dependent claims 24, 25, and 27-29, these claims include additional features such 
as a "first database", a "number of candidate image samples", a "Viterbi search", and "talking- 
head animation". Each of the claimed steps in these claims appear to be operations that may be 
performed manually without the use of a machine, i.e. one may perform a Viterbi search using 
pen and paper to manually write-out calculations. In addition, a database does not necessarily 
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have to be defined in terms of an electronic computer database. For example, one may have an 
organization or collection of items written down on papers and this would still be considered a 
database of information. 

Claims 30-32, 34, and 35 are directed to non-functional descriptive material because it merely 
cites an animation of an object which is essentially image data. Nonfunctional descriptive 
material that does not constitute a statutory process, machine, manufacture or composition of 
matter and should be rejected under 35 U.S.C. § 101. Certain types of descriptive material, such 
as music, literature, art, photographs and mere arrangements or compilations of facts or data, 
without any functional interrelationship is not a process, machine, manufacture or composition of 
matter. Thus, the claimed subject matter is nonstatutory. 

Claim Rejections - 35 USC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

2. Claims 22-25, 27, 29-32, and 34 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ezzat et al. (NPL document, "Visual Speech Synthesis by Morphing 
Visemes", herein referred to as "Ezzat") in view of Jiang et al. (NPL document, "Visual Speech 
Analysis with Application to Mandarin Speech Training", herein referred to as "Jiang") in view 
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of Cox et al. (NPL Doc, "Speech and language processing for next-millennium communications 
services"). 

As per claims 22, 23, and 30, Ezzat teaches the claimed "selecting" step on top of 1 st 
column on pg. 5 1 and states: 

"there are many intermediate frames that lie between the chosen viseme images . . . 
Consequently, we compute a series of consecutive optical flowvectors between each 
intermediate image and its successor, and concatenate them all into one large flow vector 
that defines the global transformation between the chosen visemes". (emphasis added) 

And states in the abstract: 

we are able to synchronize the visual speech stream with the audio speech stream, and 
hence give the impression of a photorealistic talking face. 

(emphasis added) 

Here, the visemes represent a generic facial image that can be use to describe a particular 
sound and the flowvectors which contain visual and sound features are used in conjunction with 
the visemes. 

Ezzat does not explicitly teach the claimed "obtaining" step. Jiang teaches the claimed 

"obtaining" step by stating in the abstract: 

At each frame, region of interest is identified and 
key information is extracted. The preprocessed acoustic 
and visual information are then fed into a modular TDNN 
and combined for visual speech analysis, (emphasis added) 

states on (pg. 1 14, 4.2 Acoustic and Visual Input Representation, 1st paragraph): 

For acoustic data representation, we have followed 
the well-established approach to apply FFT on the Hamming 
windowed speech data to get 16 Melscale Fourier coefficients as 
input to the Acoustic input Layer. For visual data representation, 
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we have performed the lip-tracking and feature points extraction 
task by applying our 2D multi-state lip shape model. Then we 
use both the color profile of the feature points on external and 
internal boundaries and position and movement of lip boundaries 
for feature extraction using principle component analysis (PCA). 
The extracted feature vectors are then fed to the Visual Input 
Layer, (emphasis added) 

Here, the Jiang teaches feature vectors (target feature vector) and teaches of visual data 
(visual features) and acoustic information (non-visual information). 

It would have been obvious to one of ordinary skill in the art at the time of invention to 
combine Ezzat with Jiang in order to provide more analysis data for the speech synthesis process. 
Ezzat is modified by Jiang by applying the target feature vectors to the phonemes and visemes as 
used in Ezzat for each frame. For example, the target feature vector may be used in the output 
frames (labeled "Video") shown in figure 1 of Ezzat. 

Ezzat does not explicitly teach the claimed "unit selection process". 
Cox teaches the claimed: 

Unit selection process in which a longest possible candidate image sample is selected (in figure 
2 and bottom of 2 nd col on page 1318, "There are two good reasons why the method of unit 
selection synthesis is capable of producing customer-quality or even natural-quality speech 
synthesis. First, on-line selection of speech segments allows for longer units (whole words, 
potentially even whole sentences) to be used in the synthesis if they are found in the inventory . ". 
In addition, the references teaches of image samples, towards the middle of the 1 st col on page 
1319, "synthesized using photo-realistic two-dimensional image technologies (sample-based 



Application/Control Number: 10/662,550 Page 7 

Art Unit: 2628 

VTTS) " where the unit selection process is used with these samples because there is a 
correspondence between the audio and visual TTS). 

It would have been obvious to one of ordinary skill in the art at the time of invention to combine 
Ezzat, Jiang, and Cox. Cox teaches one advantage of the combination (bottom of 2nd col on 
page 1318, "There are two good reasons why the method of unit-selection synthesis is capable of 
producing customer-quality or even natural-quality speech synthesis) where this can improve the 
output of Ezzat. Ezzat is modified by Cox by applying the unit selection process to select the 
best image candidate sample or viseme from the collection of possible samples (such as shown in 
figures 4 and 5 of Ezzat) in Ezzat. 

As per claims 24-25, and 31-32, Ezzat teaches the claimed "selecting . . . using a 

comparison of a combination of visual features and non-visual features with the target feature 

vector" by stating on pg. 47, 2 nd col, 2 nd paragraph: 

For any input text, we determine the appropriate sequence of viseme morphs to make, as 
well as the rate of the transformations by utilizing the output of the natural language 
processing unit (emphasis added) 

In order to determine the appropriate sequence, the system performs a comparison of visual and 
non- visual features with a given target vector in order to produce the output as stated. In this 
case, the visual features are the video images of facial movements of the mouth (for example see 
figures 4 and 5). The non-visual features are the sounds that correspond to the facial movements 
(for example what sound a given syllable makes such as a phoneme, see figure 3; also, see the 
bottom of the 2 nd col on page 47). Further, this construction process of an appropriate sequence 
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of viseme morphs would require selecting candidate image samples where these samples could 
be used to transition between through transformation. 

Ezzat teaches the claimed compiling by teaching of concatenation (see quote from top of 
1 st column on pg. 51 above). 

As per claim 27 and 34, Ezzat teaches the claimed first database by teaching of recording 
and collecting one image per English phoneme (bottom of 1 st column on pg. 47 under "Corpus 
and Viseme Acquisition", also see figure 2). 

Ezzat teaches the claimed second and third database by teaching of "Flow database" (pg. 
54, 2 nd column), which contain optical flow vectors which specify transition data between 
visemes (includes visual data and includes storing non-visual data i.e. sound transitions). 

As per claim 29, Ezzat teaches the claimed first database in figure 2, the claimed second 
database and the claimed third database on pg. 54, 2 nd column under "Flow database" where this 
database is formed to specify visual and non-visual data between animation transitions (frames). 

3. Claims 28 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ezzat in 
view of Jiang in further view of Cox in further of view of Brand (NPL Document, "Voice 
Puppetry", herein referred to as "Brand"). 

As per claims 28 and 35, Ezzat does not teach the claimed limitations. 

Brand teaches the claimed "selecting ... a number of candidates" and the claimed 
"Viterbi search" by stating on the bottom half of the 1 st col on pg. 25: 
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The Viterbi sequence, while most likely, may only represent a small fraction of the total 
probability mass — there may be thousands of slightly different state sequences that 
are nearly as likely. If this were to happen in the voice puppet, V would be a very poor 
representation of the relevant information 
in the audio, and the animation quality would suffer greatly. 

. . . These problems are virtually banished with entropically estimated models because 
entropy minimization concentrates the probability mass on the optimal Viterbi 
sequence, (emphasis added) 

Brand teaches the claimed calculating concatenation cost by stating on pg. 26, very 

bottom of 1 st col and very top of 2 nd col: 

We quantified this with a squared error measure of divergence between groundtruth (x) 
and reconstructed (y) facial motion vectors, weighted to penalize motions in the wrong 
direction, (emphasis added) 



It would have been obvious to one of ordinary skill in the art at the time of invention to combine 
Brand with the combinable system of Ezzat, Jiang, and Cox. Brand teaches the advantage of 
using an optimal Viterbi sequence with a large number of state sequences (candidates) to reduce 
the size to the most optimal ones in order to remove poor animation quality (1 st col on pg. 25 see 
quote above). Ezzat is modified by Brand by incorporating the Viterbi search of Brand to select 
the best viseme or image sample to use with the collection of samples in Ezzat. 



Response to Arguments 

1 . Applicant's arguments filed 1 1/26/2008 have been fully considered but they are not 
persuasive. 



Applicant argues: 
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Applicants submit an affidavit under 37 C.F.R. § 1.131 to establish reduction to practice before 
the August 2000 publication date of Cox et al. in Proceedings of the IEEE, Vol. 88, No. 8, and 
diligence until the filing date. 

(upper middle of page 2 in filed response, emphasis added). 

The examiner respectfully maintains that this statement itself is confusing and not correct. The 
statement is confusing as to exactly which way the applicant intends to prove prior inventorship. 
As quoted by the applicant at the bottom of page 2 in the remarks, 37 CFR 1 . 13 1(b) 



The showing of facts shall be such, in character and weight, as to 
establish reduction to practice prior to the effective date of the 
reference, or conception of the invention prior to the effective date 
of the reference coupled with due diligence from prior to said date 
to a subsequent reduction to practice or to the filing of the 
application 
(emphasis added) 

According to the rule, there are three separate ways of showing prior inventorship. The 
following chart also illustrates these three ways more clearly: 
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Three ways to show prior 
inventorship 

Iff&etto date 
of reference 




The next chart illustrates the facts and dates supplied by the applicant thus far from their affidavit 
and remarks: 
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1 ?iree ways to show prior 
inventorship 



At least J 



WMmMm date 
of reference 



* f 



Cox 102(a) 
Aug 2000 



+ 



[1] and m 



11! 



Parent App 
Mar 29, 2001 



(Constructive 



to practice} 



j nt appl approved Mar 17, 2000 
v x - o 0, receipt Apr 12, 2000 



According to the applicant's remarks filed, the applicant established reduction to practice before 
the August 2000 (upper middle of page 2 in filed response). This would imply that the applicant 
intends use method I in the chart above for showing prior invention. However, also according to 
the applicant's remarks filed, the applicant stated that they are showing diligence until the filing 
date. This would imply that the applicant intends to use either method II or III in the chart 
above. 
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However, according to the chart shown, one cannot have both a reduction to practice 
before the August 2000 publication and diligence, because diligence is defined, within the 
context of 37 CFR 1 . 1 3 1 , to be a time period sometime before the August 2000 publication to a 
later date of reduction to practice. 

Since, it unclear which method is being applied, the examiner will consider each 
separately. First, for method I, the evidence would have to provide that the applicant established 
reduction to the practice sometime before the Aug 2000 Cox publication. According to the 
MPEP, establishing actual reduction to practice requires the need of a real-world physical 
construction of the invention. For example, sec MPEP section 2138.05 in the section titled: "II. 
REQUIREMENTS TO ESTABLISH ACTUAL REDUCTION TO PRACTICE". The following 
is stated: "The same evidence sufficient for a constructive reduction to practice may be 
insufficient to establish an actual reduction to practice, which requires a showing of the invention 
in a physical or tangible form that shows every element of the count. Wetmore v. Quick, 536 
F.2d 937, 942, 190 USPQ 223, 227 (CCPA 1976)." 

The applicant's affidavit dated 1 1/26/2008, states that a draft disclosure was written and 
completed around Mar to Apr of 2000 (in exhibits II and III). However, as stated above, the 
"The same evidence sufficient for a constructive reduction to practice may be insufficient to 
establish an actual reduction to practice". The filing of an actual patent with a disclosure is a 
constructive reduction to practice. The disclosure itself does not necessarily mean that actual 
reduction to practice has occurred. Since a draft disclosure is a written document stating how an 
invention works, it does not in-itself show "the invention in a physical or tangible form that 
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shows every element of the count". Thus, based on the information provided thus far, the 
affidavit does not show the requirements for method I of prior inventorship. 

Next, one must look at the case in regards to the remaining methods of II and III for 
showing prior inventorship. Since, the affidavit does not provide any evidence of a physical or 
tangible form of the invention in the time period after the Aug 2000 Cox publication to the time 
of filing of the parent application, it appears that method II does not apply in this case (see the 
chart above for an illustration using timelines). 

Method III does not require an actual reduction to practice but rather a constructive 
reduction to practice (the filing of a patent application). The applicant, in this case, does have 
proof that a constructive reduction to practice occurred on Mar 29, 2001 (the filing of the parent 
patent application). Thus, the additional requirement of diligence must be considered. 

In particular, the requirement of diligence must be considered from the time period of 
right before Aug 2000 to Mar 29, 2001 according to method III. The applicant discussed 
diligence from the bottom of the page 3 to the top of page 5 in the remarks. The applicant 
correctly points out that the time period in question for diligence is about 8 months (at the top of 
page 4 in the remarks). The applicant argues that the present application is 20 pages (counting 
drawings) and includes 21 claims, and that during the 8 months in question, reasonable diligence 
had been applied (bottom of page 4 in remarks). 

However, the remarks and affidavit are absent of details in this particular regard. For 
example, while exhibits II and III of the affidavit establish that the draft disclosure was sent to 
Wendy Koba with receipt, the affidavit itself is absent any information of what occurred after 
Mar- Apr 2000. In other words, the affidavit does not at all address the 8 month time period of 
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diligence. The applicant remarks state that Wendy Koba started working diligently on the 
application starting on July 31, 2000 (bottom of page 4 in remarks). Applicant only provides a 
conjecture; i.e. "For example, suppose Wendy Koba could not begin working on the application 
up until July 30, 200, and started working diligently to prepare and file the application starting on 
July 31, 2000." Applicant further makes the statement that Wendy Koba prepared this 
application with reasonable diligence without any supporting evidence. 

In regards to the situation mentioned above, if one has a draft copy of the disclosure 
already, it is unclear as to why it would take at least 8 additional months to prepare the parent 
application from a draft disclosure. The remarks and affidavit do not provide this information. 
In addition, it is not up to the office or examiner as to guess whether diligence during the 8- 
months time period occurred. For example, one cannot tell whether Wendy Koba had additional 
other work during this time period, or whether that work is related to the application in question. 
The applicant also provides case law in support of preparing an application over a time period of 
10 months; however, the case law itself does not provide actual facts of what occur for this 
particular application. Thus, it appears that the applicant has not met the conditions for method 
III of showing prior inventorship because facts are missing or not provided during the time 
period of diligence. 

Also consider the following from the MPEP: 

MPEP 715.07(a) Diligence 

Where conception occurs prior to the date of the reference, 
but reduction to practice is afterward, it is not enough merely to 
allege that applicant or patent owner had been diligent. Ex parte 
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Hunter, 1889 CD. 218, 49 O.G. 733 (Comm'r Pat. 1889). Rather, 
applicant must show evidence of facts establishing diligence. 

MPEP 2138.06 "Reasonable Diligence" 

DILIGENCE REQUIRED IN PREPARING AND FILING 
PATENT APPLICATION 

The diligence of attorney in preparing and filing patent 
application inures to the benefit of the inventor. Conception was 
established at least as early as the date a draft of a patent 
application was finished by a patent attorney on behalf of the 
inventor. 

Conception is less a matter of signature than it is one of 
disclosure. Attorney does not prepare a patent application on 
behalf of particular named persons, but on behalf of the true 
inventive entity. Six days to execute and file application is 
acceptable. Haskell v. Coleburne, 671 F.2d 1362, 213 USPQ 192, 
195 (CCPA 1982). See also Bey v. Kollonitsch, 866 F.2d 1024, 
231 USPQ 967 (Fed. Cir. 1986) (Reasonable diligence is all that 
is required of the attorney. Reasonable diligence is established 
if attorney worked reasonably hard on the application during 
the continuous critical period. If the attorney has a reasonable 
backlog of unrelated cases which he takes up in chronological 
order and carries out expeditiously, that is sufficient. Work on 
a related case(s) that contributed substantially to the ultimate 
preparation of an application can be credited as diligence.). 

Applicant has not shown at least the highlighted part. 



Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to DANIEL F. HAJNIK whose telephone number is (571)272-7642. 
The examiner can normally be reached on Mon-Fri (8:30A-5:00P). 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Xiao Wu can be reached on (571) 272-7761 . The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

/Daniel Hajnik/ 
Patent Examiner 
Art Unit 2628 

/XIAO M. WU/ 



Supervisory Patent Examiner, Art Unit 2628 



